Import for ADE20K dataset #400

sizov-kirill · 2021-08-05T03:58:22Z

Summary

Resolved #399.

How to test

Checklist

I submit my changes into the develop branch
I have added description of my changes into CHANGELOG
I have updated the documentation accordingly
I have added tests to cover my changes
I have linked related issues

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
I have updated the license header for each file (see an example below)

# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

docs/formats/ade20k2020_user_manual.md

datumaro/plugins/ade20k2020_format.py

tests/test_ade20k2020_format.py

docs/formats/ade20k2020_user_manual.md

docs/formats/ade20k2017_user_manual.md

zhiltsov-max

Could you check if the imported dataset can be converted into some other format? Pascal VOC or other.

… into sk/ade20k

zhiltsov-max · 2021-08-19T10:34:59Z

datumaro/plugins/ade20k2020_format.py


+            if Ade20k2020Path.MASK_IMAGE_PATTERN.search(item_id):


Probably, only the basename should be checked. Also, consider using re.fullmatch to match the whole name.

Okay, I will change the item_id to the basename, but I don't think that we should use re.,fullmatch here, because we don't know full name for instance mask (that should be skipped). We only recognize it after parsing the annotation file.

sizov-kirill · 2021-08-20T08:05:59Z

Could you check if the imported dataset can be converted into some other format? Pascal VOC or other.

I checked converting with VOC, COCO and KITTI formats. Conversion to COCO failed, but I think it's no related with this PR, the problem occurs here:

datumaro/datumaro/plugins/coco_format/converter.py

Lines 232 to 234 in 935fb0e

    
           masks = (m.image for m in masks) 
        
           if mask is not None: 
        
               masks += chain(masks, [mask])

masks has type generator, but then we use += between generator and itertools.chain Python doesn't support such concatenation. Or I'm wrong?

zhiltsov-max · 2021-08-20T09:14:25Z

datumaro/plugins/ade20k2020_format.py

+    MASK_PATTERN = re.compile(r'''\w+_seg\.\w+
+        | \w+_parts_\d+\.\w+
+        | instance_\w+\.\w+
+    ''', re.VERBOSE)


Isn't \w+ too restrictive? It won't accept digits and special characters like - and _.

Okay, I'll add - and spaces, regarding digits and underscores In the docs says that \w accepts them.

zhiltsov-max · 2021-08-20T09:15:39Z

masks has type generator, but then we use += between generator and itertools.chain Python doesn't support such concatenation. Or I'm wrong?

Yes, looks like a bug in COCO. Can you fix it?

sizov-kirill · 2021-08-20T09:24:31Z

masks has type generator, but then we use += between generator and itertools.chain Python doesn't support such concatenation. Or I'm wrong?

Yes, looks like a bug in COCO. Can you fix it?

Yes, I'll fix it in new PR.

zhiltsov-max · 2021-08-20T13:08:16Z

datumaro/plugins/ade20k2017_format.py

+    MASK_PATTERN = re.compile(r'''[\w|\s|-]+_seg\.\w+
+        | [\w|\s|-]+_parts_\d+\.\w+
+    ''', re.VERBOSE)


Suggested change

MASK_PATTERN = re.compile(r'''[\w|\s|-]+_seg\.\w+

| [\w|\s|-]+_parts_\d+\.\w+

''', re.VERBOSE)

MASK_PATTERN = re.compile(r'''

.+_seg

| .+_parts_\d+

''', re.VERBOSE)

zhiltsov-max · 2021-08-20T13:09:54Z

datumaro/plugins/ade20k2020_format.py

+    MASK_PATTERN = re.compile(r'''[\w|\s|-]+_seg\.\w+
+        | [\w|\s|-]+_parts_\d+\.\w+
+        | instance_[\w|\s|-]+\.\w+


zhiltsov-max · 2021-08-20T13:10:26Z

datumaro/plugins/ade20k2020_format.py

+        for image_path in sorted(images):
+            item_id = osp.splitext(osp.relpath(image_path, path))[0]
+
+            if Ade20k2020Path.MASK_PATTERN.fullmatch(osp.basename(image_path)):


Suggested change

if Ade20k2020Path.MASK_PATTERN.fullmatch(osp.basename(image_path)):

if Ade20k2020Path.MASK_PATTERN.fullmatch(osp.basename(image_id)):

zhiltsov-max · 2021-08-20T13:10:41Z

datumaro/plugins/ade20k2017_format.py

+        for image_path in sorted(images):
+            item_id = osp.splitext(osp.relpath(image_path, path))[0]
+
+            if Ade20k2017Path.MASK_PATTERN.fullmatch(osp.basename(image_path)):


Suggested change

if Ade20k2017Path.MASK_PATTERN.fullmatch(osp.basename(image_path)):

if Ade20k2017Path.MASK_PATTERN.fullmatch(osp.basename(image_id)):

kirill.sizov added 3 commits August 5, 2021 06:52

Add import for ade20k

e0ea964

Add test

9f63add

Sort imports

cd6c3e9

sizov-kirill changed the title ~~Import for ADE20K dataset~~ [WIP] Import for ADE20K dataset Aug 5, 2021

kirill.sizov added 13 commits August 5, 2021 12:40

Rename some files and classes

bc3af60

Sort image paths before reading them

2f0b19e

Use exception instead warning

087e725

Sort subsets list

cee351f

Fix test

e2e374d

Use z_order

42a501e

Add documentation file

d1791ec

Update user manual

11bd0db

Fix remark issue

5affcb1

Fix test

d916f06

Use lazy mask

269aaea

Sort imports

e237762

Delete whitespace

b0985e5

sizov-kirill requested a review from zhiltsov-max August 6, 2021 11:24

sizov-kirill changed the title ~~[WIP] Import for ADE20K dataset~~ Import for ADE20K dataset Aug 6, 2021

kirill.sizov added 6 commits August 16, 2021 11:04

Update tests

08f04a4

Support for 2020 year version

194c35a

Sort imports

8c3b8ab

Update documentation

7c0d47e

Update documentation

4d34ce3

Remove long lines

bbc28fd

zhiltsov-max reviewed Aug 16, 2021

View reviewed changes

docs/formats/ade20k2020_user_manual.md Outdated Show resolved Hide resolved

zhiltsov-max reviewed Aug 16, 2021

View reviewed changes

docs/formats/ade20k2020_user_manual.md Outdated Show resolved Hide resolved

zhiltsov-max reviewed Aug 16, 2021

View reviewed changes

datumaro/plugins/ade20k2020_format.py Outdated Show resolved Hide resolved

zhiltsov-max reviewed Aug 16, 2021

View reviewed changes

tests/test_ade20k2020_format.py Show resolved Hide resolved

zhiltsov-max reviewed Aug 16, 2021

View reviewed changes

docs/formats/ade20k2020_user_manual.md Outdated Show resolved Hide resolved

kirill.sizov added 2 commits August 18, 2021 15:05

Search files more carefully

d370d6b

Use tree output documentation examples

a3a54f5

sizov-kirill requested a review from zhiltsov-max August 18, 2021 12:28

Update ade20k2017_user_manual.md

a30bace

zhiltsov-max reviewed Aug 18, 2021

View reviewed changes

docs/formats/ade20k2017_user_manual.md Outdated Show resolved Hide resolved

zhiltsov-max reviewed Aug 18, 2021

View reviewed changes

kirill.sizov added 6 commits August 19, 2021 11:38

Fix instance mask, add id

4651ec8

Update tests

f2558d6

Update docs

eb8e7a2

Merge branch 'sk/ade20k' of https://github.com/openvinotoolkit/datumaro…

13b0604

… into sk/ade20k

Search all image extensions

3ac08c3

Update docs

7c73bd4

zhiltsov-max reviewed Aug 19, 2021

View reviewed changes

kirill.sizov added 4 commits August 20, 2021 11:28

Use fullmatch and basename

d261621

Fixed points ordering for polygons

ffaa229

Add polygon to test

10b96cb

Delete unused imports

7e4f9ee

zhiltsov-max reviewed Aug 20, 2021

View reviewed changes

Allow spaces and dash in image name

8fb387b

zhiltsov-max reviewed Aug 20, 2021

View reviewed changes

kirill.sizov added 4 commits August 20, 2021 16:18

Update regexp

12c69a0

Update regexp

4eec823

Merge branch 'develop' into sk/ade20k

9a973ec

Update changelog

db324c2

zhiltsov-max approved these changes Aug 20, 2021

View reviewed changes

zhiltsov-max merged commit f1df870 into develop Aug 20, 2021

zhiltsov-max deleted the sk/ade20k branch August 20, 2021 14:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Import for ADE20K dataset #400

Import for ADE20K dataset #400

sizov-kirill commented Aug 5, 2021 •

edited

Loading

zhiltsov-max left a comment

zhiltsov-max Aug 19, 2021

sizov-kirill Aug 19, 2021 •

edited

Loading

sizov-kirill commented Aug 20, 2021

zhiltsov-max Aug 20, 2021

sizov-kirill Aug 20, 2021

zhiltsov-max commented Aug 20, 2021 •

edited

Loading

sizov-kirill commented Aug 20, 2021

zhiltsov-max Aug 20, 2021

zhiltsov-max Aug 20, 2021

zhiltsov-max Aug 20, 2021

zhiltsov-max Aug 20, 2021

	if Ade20k2020Path.MASK_PATTERN.fullmatch(osp.basename(image_path)):
	if Ade20k2020Path.MASK_PATTERN.fullmatch(osp.basename(image_id)):

	if Ade20k2017Path.MASK_PATTERN.fullmatch(osp.basename(image_path)):
	if Ade20k2017Path.MASK_PATTERN.fullmatch(osp.basename(image_id)):

Import for ADE20K dataset #400

Import for ADE20K dataset #400

Conversation

sizov-kirill commented Aug 5, 2021 • edited Loading

Summary

How to test

Checklist

License

zhiltsov-max left a comment

Choose a reason for hiding this comment

zhiltsov-max Aug 19, 2021

Choose a reason for hiding this comment

sizov-kirill Aug 19, 2021 • edited Loading

Choose a reason for hiding this comment

sizov-kirill commented Aug 20, 2021

zhiltsov-max Aug 20, 2021

Choose a reason for hiding this comment

sizov-kirill Aug 20, 2021

Choose a reason for hiding this comment

zhiltsov-max commented Aug 20, 2021 • edited Loading

sizov-kirill commented Aug 20, 2021

zhiltsov-max Aug 20, 2021

Choose a reason for hiding this comment

zhiltsov-max Aug 20, 2021

Choose a reason for hiding this comment

zhiltsov-max Aug 20, 2021

Choose a reason for hiding this comment

zhiltsov-max Aug 20, 2021

Choose a reason for hiding this comment

sizov-kirill commented Aug 5, 2021 •

edited

Loading

sizov-kirill Aug 19, 2021 •

edited

Loading

zhiltsov-max commented Aug 20, 2021 •

edited

Loading