Add validator for classification & detection tasks

[seongjun-park-dl]

Summary

This PR includes:

• Validator for classification and detection tasks.
• CLI support for validator.

How to test

python -m unittest -v tests/test_validator.py

Checklist

• I submit my changes into the develop branch
• I have added description of my changes into CHANGELOG
• I have updated the documentation accordingly
• I have added tests to cover my changes
• I have linked related issues)

License

• I submit my code changes under the same MIT License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below)

# Copyright (C) 2020 Intel Corporation
#
# SPDX-License-Identifier: MIT

[zhiltsov-max]

1. Generally, looks ok. I've tried to validate VOC and this is what I got:

"summary": {
        "errors": 319372,
        "warnings": 81169
    },

2. The other thing I found is:

{
            "anomaly_type": "MissingBboxAnnotation",
            "description": "Item needs one or more bounding box annotations, but not found.",
            "item_id": "2007_000032",
            "severity": "warning"
        },

I think, it would be useful to add item subset as well (or group results by subsets)

3. This check

{
            "anomaly_type": "FarFromLabelMean",
            "description": "Bounding box annotation '11' in the item has a value of bounding box 'y' that is too far from the label average. (mean of 'person' label: 108.82, got '286.0').",
            "item_id": "2008_004948",
            "severity": "warning"
        },

looks quite strange for 'x' and 'y' bbox attributes - why images in the dataset must have anything common for them? 'width' and 'height' are more or less understandable, but it might be good to check against clusters, instead of a single value of average. Another thought is to allow setting acceptable boundaries.
'long' and 'short' are hard to understand in bbox context.

4. This check

{
            "anomaly_type": "UndefinedAttribute",
            "description": "Item has the attribute 'difficult' for the label 'person' which is not defined in metadata.",
            "item_id": "2011_002650",
            "severity": "error"
        },

should consider attributes in LabelCategories.attributes as general, i.e. applicable to all labels. Label-specific ones are written in each label.

5. As a further step, I'd recommend think of providing an option to control enabled checks.

[jihyeonyi]

For the 2nd comment (grouping missing bbox cases), your idea seems really nice but I don't think we can apply the groupping function at this PR.
For now, the concept of validator is just listing all the warnings verbosely, so the user should group or filter items by parsing the validation results file.
And I can't think of a nice interface? process? to do that.
We can implement this as a filtering function, transformer, or utility function. I don't know which is the best.

[jihyeonyi]

For 3rd comment, I agree with you.
And for the short(=min(w,h)) and long(=max(w,h)), some researchers think they are important for the detection task.
So we include those terms.

For 4rd comment, we didn't know that. Seongjun would fix that.

And for the last comment, we'll consider that later, because it is a design issue.

[seongjun-park-dl]

I've addressed 3rd and 4th comments in my most recent commit.

[zhiltsov-max]

For now, the concept of validator is just listing all the warnings verbosely, so the user should group or filter items by parsing the validation results file.

I mean, it would simplify/clarify things, if error messages contained a "subset" field, because a single id can (theoretically) be found in several subsets. And even if subsets do not have same item ids, it still makes the process of error localization simpler.

{
"anomaly_type": "...",
"description": "...",
"item_id": "2007_000032",
"subset": "train",
"severity": "..."
},

[jihyeonyi]

For now, the concept of validator is just listing all the warnings verbosely, so the user should group or filter items by parsing the validation results file.

I mean, it would simplify/clarify things, if error messages contained a "subset" field, because a single id can (theoretically) be found in several subsets. And even if subsets do not have same item ids, it still makes the process of error localization simpler.

{
"anomaly_type": "...",
"description": "...",
"item_id": "2007_000032", "subset": "train",
"severity": "..."
},

Oh, I can understand now. It's not difficult to add "subset" information and it seems useful.

[seongjun-park-dl]

For now, the concept of validator is just listing all the warnings verbosely, so the user should group or filter items by parsing the validation results file.

I mean, it would simplify/clarify things, if error messages contained a "subset" field, because a single id can (theoretically) be found in several subsets. And even if subsets do not have same item ids, it still makes the process of error localization simpler.

{
"anomaly_type": "...",
"description": "...",
"item_id": "2007_000032", "subset": "train",
"severity": "..."
},

Added the subset field to the error messages!