Add kolektor dataset

[Ravindu987]

Description

• Work in progress

• Added the dataset classes, dataloaders and other necessary functions to load Kolektor Dataset

• Due to the nature of the dataset, used unconventional methods which needs to be discussed

• Requires ImageIO and Numpy libraries

• Fixes [Task]: Implement Kolektor dataset #964

Changes

• Bug fix (non-breaking change which fixes an issue)
• Refactor (non-breaking change which refactors the code base)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist

• My code follows the pre-commit style and check guidelines of this project.
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing tests pass locally with my changes
• I have added a summary of my changes to the CHANGELOG (not for minor changes, docs and tests).

[ashwinvaidya17]

Thanks for the efforts! I understand this is WIP, I've still gone ahead and added a few comments. Also, can you also accress the issues raised by Codacy.

[ashwinvaidya17]

This is missing prepare_data method. Have a look at

anomalib/src/anomalib/data/mvtec.py

Line 258 in 27adbae

def prepare_data(self) -> None:

to see an example implementation

[Ravindu987]

Hi @ashwinvaidya17
Thank you very much for your feedback.
In the latest commits I resolved three of the issues you raised.

Regarding the prepare_dataset there's an issue. The download link for the dataset doesn't provide an extension for the downloaded file. Instead it sends the extension via the response header. This works well with a browser but the urlretrieve method used in the download_and_extract method downloads the file without extension and therefore cannot extract.

We have two options to resolve this

1. Instead of download_and_extract use a custom function to download and set the extension.
2. Update the download_and_extract function to receive the response header and set the extension. If we choose this we'll have to update other make_dataset methods accordingly.

Please let me know your thoughts on this.

Also as I have informed @samet-akcay , I had to pause the work on this until I get a green light on the current logical implementation ( As you might have seen, I had to classify images on the runtime instead of loading from directories because of the nature of the dataset). Please let me know whether the logic is good to go so that I can finish up.

[Ravindu987]

Hey guys
I've gone ahead and completed writing docstrings and implemented a custom function to download and extract the dataset.
Since I've tested it locally I removed the WIP tag. Please have a look.

Could you refer me a documentation on the test convention used here. Once I get an insight on this I can finish it as well.

Thanks

[samet-akcay]

Thanks @Ravindu987 for your great contribution! I have a comment regarding the use of download and extract function.

[samet-akcay]

If you define DOWNLOAD_INFO variable as given below

DOWNLOAD_INFO = DownloadInfo(
    name="kolektor",
    url="https://go.vicos.si/kolektorsdd",
    hash="2b094030343c1cd59df02203ac6c57a0",
)

Then it would be possible to use anomalib.data.download.download_and_extract function as:

download_and_extract(self.root, DOWNLOAD_INFO)

This would be inline with the rest of the datamodule implementations. For reference, here are the some of the implementations:
mvtec-ad, avenue, btech, mvtec-3d...

[Ravindu987]

This links with the previous issue which makes the download_and_extract function not usable here.

[Ravindu987]

Thank you very much @samet-akcay for the review. I resolved one requested change and answered your questions as well. Please let me know your thoughts and any further changes.

[samet-akcay]

lgtm, thanks a lot for your contribution!

[ashwinvaidya17]

Thanks again for your efforts!

[Ravindu987]

Thanks for the review guys.
The Codacy Static Code Analysis fails because the init function in the DataModule class has too many arguments. I see no way around this as other implementations have the same issue.
Not sure why the Tox checks are failing. When I ran all pre-merge checks locally they all passed. Any ide on this?

[samet-akcay]

Thanks for the review guys. The Codacy Static Code Analysis fails because the init function in the DataModule class has too many arguments. I see no way around this as other implementations have the same issue. Not sure why the Tox checks are failing. When I ran all pre-merge checks locally they all passed. Any ide on this?

It's a problem with our CI that we need to fix. You don't need to do anything else at this point. Thank again for your contribution

[Ravindu987]

Thanks for the review guys. The Codacy Static Code Analysis fails because the init function in the DataModule class has too many arguments. I see no way around this as other implementations have the same issue. Not sure why the Tox checks are failing. When I ran all pre-merge checks locally they all passed. Any ide on this?

It's a problem with our CI that we need to fix. You don't need to do anything else at this point. Thank again for your contribution

Noted with thanks. Hope to contribute more in the future.